WIP: FEATURE: Rework CR CatchUp mechanism #4988

bwaidelich · 2024-04-11T15:56:44Z

related: #4746

WIP because:

Proper(er) mechanism to prevent multiple catchup runs (maybe integrate TASK: Prevent multiple catchup runs #4751)?
Mechanism to skip projections that are not up-to-date and don't trigger hooks on replay/individual catchup – (maybe integrate with Replace CatchUpHooks with ContentRepositoryHooks #4992)
needs to be properly coordinated with other pending PRs like !!! TASK: Interdimensional relatives for node move #4993, WIP: FEATURE: Integrate ContentStream projection into ContentGraph #4980, FEATURE: ContentGraphAdapter for write side access #4979
update neos docs to not promote block see https://docs.neos.io/guide/contributing-to-neos/event-sourced-content-repository/blocking-until-a-command-is-executed and related

as discussed in todays weekly

and introduce `neos/contentrepository-dbaltools` package

…b.com/neos/neos-development-collection into feature/4746-rework-catchup-mechanism

...and use `Doctrine\DBAL\Connection` directly

bwaidelich · 2024-04-16T13:13:28Z

Neos.ContentGraph.DoctrineDbalAdapter/Tests/Behavior/Features/Bootstrap/FeatureContext.php

@@ -44,7 +44,6 @@ public function __construct()
        self::bootstrapFlow();
        $this->contentRepositoryRegistry = $this->getObject(ContentRepositoryRegistry::class);

-        $this->setupCRTestSuiteTrait();


no longer needed, see

neos-development-collection/Neos.ContentRepository.TestSuite/Classes/Behavior/Features/Bootstrap/CRTestSuiteTrait.php

Line 88 in 9ee2fc8

protected function setupCRTestSuiteTrait(): void

..to avoid status from showing a diff everytime

...for now

bwaidelich · 2024-04-19T13:18:32Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+    {
+        $store = new SemaphoreStore();
+        $factory = new LockFactory($store);
+        $lock = $factory->createLock('catchup');


TODO: This lock needs to be CR specific (i.e. include $this->id)

bwaidelich · 2024-04-19T13:20:44Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

@@ -153,6 +159,67 @@ public function projectionState(string $projectionStateClassName): ProjectionSta
        throw new \InvalidArgumentException(sprintf('A projection state of type "%s" is not registered in this content repository instance.', $projectionStateClassName), 1662033650);
    }

+    public function catchUpProjections(): void
+    {
+        $store = new SemaphoreStore();


This has to become an external dependency and/or coordinated with #4751

nezaniel · 2024-04-21T09:48:01Z

This already looks very promising and I like the direction this is taking!
What part is still missing / incomplete? Just the new catchup implementation, right?

bwaidelich · 2024-04-21T10:38:34Z

This already looks very promising and I like the direction this is taking!

Thanks for the feedback, appreciated!

What part is still missing / incomplete?

I added some more lines to the task list above

mhsdesign

Thats a lot of stuff but looks "good" so far (considering this is WIP) ... i went through all php classes by reading except the last 20 with only the +-1 changes

mhsdesign · 2024-04-22T19:14:37Z

...pository.BehavioralTests/Tests/Functional/Feature/WorkspacePublication/setup-is-running-flag

why do we need this file now? ^^

mhsdesign · 2024-04-22T19:15:27Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+        #print_r(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS));exit;
+        #\Neos\Flow\var_dump('HANDLE ' . $command::class);


mhsdesign · 2024-04-22T19:16:48Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+        return new class {
+            /**
+             * @deprecated backwards compatibility layer
+             */
+            public function block(): void
+            {
+            }
+        };


if we were to merge this like this - which i think is not necessary - we should use an actual class so its inspectable via phpstorm :) But i dont think we need b/c layer for new code

mhsdesign · 2024-04-22T19:17:57Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

            $catchUpHook?->onBeforeEvent($event, $eventEnvelope);
            $projection->apply($event, $eventEnvelope);
+            // TODO this should happen in the inner transaction


todo over here :)

mhsdesign · 2024-04-22T19:18:20Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+            ];
+            $projectionsAndCatchUpHooks[$projectionClassName]['catchUpHook']?->onBeforeCatchUp();
+        }
+        #\Neos\Flow\var_dump('CATCHUP from ' . $lowestAppliedSequenceNumber->value);


.. lots of debug code and i guess todos?

Neos.ContentRepository.Core/Classes/Infrastructure/DbalClientInterface.php

mhsdesign · 2024-04-22T19:20:01Z

Neos.ContentRepository.Core/Classes/Infrastructure/DbalCheckpointStorage.php

i guess this is now part of CheckpointHelper? Good thing we have it in the Core and now our lower level package ^^

See #5005

in order to prevent nested transactions potentially leading to RuntimeException of A transaction is active already, can't commit events!

…tchup-mechanism

mhsdesign · 2024-05-17T08:32:11Z

i just merged 9.0 into your pr. A few notes from my side:

\ContentGraphTableNames::tableNamePrefix is now public again because its passed to the separate package DbalTools this makes ContentGraphTableNames::checkpoint obsolete as well ... should be adjusted
i started merging the .feature behat tests but at last i decided to just accept 9.0 as it was getting tedious. My assumption based on the features i have seen changed is that only the blocking part was removed. As discussed ill split this up into a separate pr either way.
- -> TASK: Remove explicit deprecated command blocking #5059
the (new - from the frankfurt sprint) shallow packages Neos.ContentRepositoryRegistry.DoctrineDbalClient Neos.ContentRepositoryRegistry.PostgresDbalClient seem to be now obsolete which is good imo (we have too much stuff first level for any new contributor. They should be removed.
Due to only having the CommandHandlingDependencies at hand we cannot pass the cr to publishEvents to block.

This change was partially ported from neos#4988 Following things have been adjusted: - Removal of the step: The graph projection is fully up to date - remove lastCommandOrEventResult - Removal of the step: The documenturipath projection is up to date - Use injected Connection in tests instead of DoctrineDbalClient (this will be fully refactored in neos#4988)

…tchup-mechanism

This change was partially ported from neos/neos-development-collection#4988 Following things have been adjusted: - Removal of the step: The graph projection is fully up to date - remove lastCommandOrEventResult - Removal of the step: The documenturipath projection is up to date - Use injected Connection in tests instead of DoctrineDbalClient (this will be fully refactored in #4988)

mhsdesign · 2024-05-18T12:26:23Z

...epository.NodeMigration/src/Transformation/AddDimensionShineThroughTransformationFactory.php

as discussed in #5061 we want to keep the result thing for now and with it the return values here.

git checkout 9.0 -- Neos.ContentRepository.Core/Classes/Service/ContentStreamPruner.php git checkout 9.0 -- Neos.ContentRepository.NodeMigration

…up-mechanism # Conflicts: # Neos.ContentGraph.DoctrineDbalAdapter/src/DoctrineDbalContentGraphProjection.php # Neos.ContentGraph.DoctrineDbalAdapter/src/Domain/Projection/ProjectionIntegrityViolationDetector.php # Neos.ContentGraph.DoctrineDbalAdapter/src/Domain/Repository/ProjectionContentGraph.php # Neos.ContentGraph.PostgreSQLAdapter/src/Domain/Projection/HypergraphProjection.php # Neos.ContentGraph.PostgreSQLAdapter/src/Domain/Projection/ProjectionHypergraph.php # Neos.ContentRepository.Core/Classes/Projection/ContentStream/ContentStreamProjection.php # Neos.ContentRepository.Core/Classes/Projection/Workspace/WorkspaceProjection.php # Neos.ContentRepositoryRegistry.DoctrineDbalClient/Configuration/Objects.yaml # Neos.ContentRepositoryRegistry.DoctrineDbalClient/Configuration/Settings.yaml # Neos.ContentRepositoryRegistry.DoctrineDbalClient/composer.json # Neos.ContentRepositoryRegistry.PostgresDbalClient/Configuration/Objects.yaml # Neos.ContentRepositoryRegistry.PostgresDbalClient/composer.json # Neos.ContentRepositoryRegistry/Classes/Command/ContentGraphIntegrityCommandController.php # Neos.ContentRepositoryRegistry/Configuration/Objects.yaml # Neos.ContentRepositoryRegistry/Configuration/Settings.yaml # Neos.Neos/Classes/PendingChangesProjection/ChangeProjection.php

mhsdesign · 2024-05-19T17:04:24Z

phpstan.neon.dist

+        - Neos.ContentRepositoryRegistry.DoctrineDbalClient
+        - Neos.ContentRepositoryRegistry.PostgresDbalClient


Suggested change

- Neos.ContentRepositoryRegistry.DoctrineDbalClient

- Neos.ContentRepositoryRegistry.PostgresDbalClient

they have been removed again, my change can thus be reverted

I'll take care of the rebasing once my three cleanups #5073 #5074 #5075 are in

github-actions bot added the 9.0 label Apr 11, 2024

WIP: FEATURE: Rework CR CatchUp mechanism

3a2fc46

bwaidelich force-pushed the feature/4746-rework-catchup-mechanism branch from 1138434 to 3a2fc46 Compare April 11, 2024 15:57

bwaidelich added 8 commits April 12, 2024 18:42

Remove react/async dependency

99cd1e0

as discussed in todays weekly

Remove react/async dependency 2/2

e8a0bdd

Cosmetic fix to satisfy linter

d2382ee

Remove unused namespace imports

7dc35fc

Replace CheckpointStorageInterface

13e1238

and introduce `neos/contentrepository-dbaltools` package

Replace CheckpointStorageInterface

4c425cd

and introduce `neos/contentrepository-dbaltools` package

Merge branch 'feature/4746-rework-catchup-mechanism' of https://githu…

b1d28f6

…b.com/neos/neos-development-collection into feature/4746-rework-catchup-mechanism

Remove DbalClientInterface and implementations

ad57a91

...and use `Doctrine\DBAL\Connection` directly

bwaidelich commented Apr 16, 2024

View reviewed changes

bwaidelich added 7 commits April 16, 2024 15:16

Update composer manifest

b817b1d

Tweak checkpoint error message

cb653fd

Re-add neos/eventstore-doctrineadapter dependency

43da4c2

Mostly cosmetic tweaks to satisfy linter

bb1c498

Make checkpoint table check a platform specific check

e62cb87

..to avoid status from showing a diff everytime

Remove notion of "sync" from composer scripts and readme

7cc13ed

Remove ProjectionCatchUpTriggerInterface and implementations

fc6ec5c

...for now

bwaidelich force-pushed the feature/4746-rework-catchup-mechanism branch from 192256e to fc6ec5c Compare April 17, 2024 15:33

bwaidelich mentioned this pull request Apr 18, 2024

WIP: Replace CatchUpHooks with ContentRepositoryHooks #4995

Draft

1 task

bwaidelich commented Apr 19, 2024

View reviewed changes

bwaidelich mentioned this pull request Apr 22, 2024

BUGFIX: Use dedicated connection for EventStore #5005

Draft

mhsdesign reviewed Apr 22, 2024

View reviewed changes

bwaidelich added 3 commits April 24, 2024 17:37

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

89eada5

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

cc3a191

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

7c8ae2c

bwaidelich added 4 commits May 1, 2024 10:43

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

4af1e3c

BUGFIX: Remove doctrine migrate and cr setup commands from behat scripts

4c68ca7

See #5005

Use dedicated connection for EventStore

7bfd941

in order to prevent nested transactions potentially leading to RuntimeException of A transaction is active already, can't commit events!

Re-use previously created event store instances

e2e8af6

bwaidelich mentioned this pull request May 6, 2024

WIP: Rework CatchUp and CatchUp hooks #5027

Closed

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

fb9b897

This was referenced May 12, 2024

TASK: WIP Prepare for synchronous content repository #5049

Merged

META: Current PR Dependency graph #5057

Open

mhsdesign added 2 commits May 17, 2024 10:23

Merge remote-tracking branch 'origin/9.0' into feature/4746-rework-ca…

082c5ca

…tchup-mechanism

TASK: Adjustments after merging 9.0 (beta10) to satisfy phpstan mostly

adf6eef

mhsdesign mentioned this pull request May 17, 2024

TASK: Remove explicit deprecated command blocking #5059

Merged

6 tasks

Merge remote-tracking branch 'origin/9.0' into feature/4746-rework-ca…

c6920ec

…tchup-mechanism

mhsdesign mentioned this pull request May 17, 2024

TASK: Remove references to CommandResult in return types #5061

Closed

6 tasks

mhsdesign reviewed May 18, 2024

View reviewed changes

bwaidelich added 4 commits May 19, 2024 12:51

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

88abe17

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

b4fe6e9

Make ContentGraphTableNames::tableNamePrefix private again

3abc8f0

mhsdesign reviewed May 19, 2024

View reviewed changes

mhsdesign mentioned this pull request May 19, 2024

TASK: Remove transaction closures from projections #5075

Merged

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

3d6edc7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: FEATURE: Rework CR CatchUp mechanism #4988

WIP: FEATURE: Rework CR CatchUp mechanism #4988

bwaidelich commented Apr 11, 2024 •

edited by mhsdesign

bwaidelich Apr 16, 2024

bwaidelich Apr 19, 2024

bwaidelich Apr 19, 2024

nezaniel commented Apr 21, 2024

bwaidelich commented Apr 21, 2024

mhsdesign left a comment •

edited

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024 •

edited

mhsdesign commented May 17, 2024 •

edited

mhsdesign May 18, 2024 •

edited

mhsdesign May 19, 2024

bwaidelich May 19, 2024

		#print_r(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS));exit;
		#\Neos\Flow\var_dump('HANDLE ' . $command::class);

		- Neos.ContentRepositoryRegistry.DoctrineDbalClient
		- Neos.ContentRepositoryRegistry.PostgresDbalClient

WIP: FEATURE: Rework CR CatchUp mechanism #4988

Are you sure you want to change the base?

WIP: FEATURE: Rework CR CatchUp mechanism #4988

Conversation

bwaidelich commented Apr 11, 2024 • edited by mhsdesign

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nezaniel commented Apr 21, 2024

bwaidelich commented Apr 21, 2024

mhsdesign left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhsdesign Apr 22, 2024 • edited

Choose a reason for hiding this comment

mhsdesign commented May 17, 2024 • edited

mhsdesign May 18, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwaidelich commented Apr 11, 2024 •

edited by mhsdesign

mhsdesign left a comment •

edited

mhsdesign Apr 22, 2024 •

edited

mhsdesign commented May 17, 2024 •

edited

mhsdesign May 18, 2024 •

edited